Support for Chinese, Lora, and ControlNet in Flux.1, plugin overview

Recently, the original team of SD developed a powerful text-to-image model, Flux.1. Such a strong model cannot afford poor support for Chinese, nor can it lack support for LoRA and ControlNet. Therefore, recently, XLabs-AI has open-sourced the LoRA and ControlNet fine-tuning scripts suitable for Flux.1-dev. In addition, Alibaba Cloud’s Bailing platform launched the optimized Chinese version of Flux on August 8, allowing users to make 1,000 free calls in the first month. Therefore, the following will provide an overview and share insights on these two points.

First, let’s outline the fine-tuning process and methods of Flux.1-dev. The entire model training is based on the deepspeed training framework.

git clone https://github.com/XLabs-AI/x-flux.
# configuration environment
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# The data format is as follows
├── images/
│    ├── 1.png
│    ├── 1.json
│    ├── 2.png
│    ├── 2.json
│    ├── ...
# json
{
    "caption": "A figure stands in a misty landscape, wearing a mask with antlers and dark, embellished attire, exuding mystery and otherworldlines"
}

accelerate launch train_flux_lora_deepspeed.py --config "train_configs/test_lora.yaml"

python3 demo_lora_inference.py \
    --repo_id XLabs-AI/flux-RealismLora \
    --prompt "A handsome girl in a suit covered with bold tattoos and holding a pistol. Animatrix illustration style, fantasy style, natural photo cinematic" --offload --name flux-dev-fp8

python3 demo_lora_inference.py \
    --repo_id XLabs-AI/flux-RealismLora \
    --prompt "A handsome girl in a suit covered with bold tattoos and holding a pistol. Animatrix illustration style, fantasy style, natural photo cinematic" \
    --offload \
    --name flux-dev-fp8

accelerate launch train_flux_deepspeed_controlnet.py --config "train_configs/test_canny_controlnet.yaml

python3 demo_controlnet_inference.py \
    --checkpoint controlnet.safetensors \
    --control_image "input_image.jpg" \
    --prompt "a bright blue bird in the garden, natural photo cinematic, MM full HD"

Those who are interested can fine-tune it themselves and give it a try. For those who want to use it directly, XLabs-AI has also open-sourced these two models, as shown below:

https://huggingface.co/XLabs-AI/flux-controlnet-canny

https://huggingface.co/XLabs-AI/flux-RealismLora

For example:

Next, we share the Chinese optimized version of Flux.1 launched on the Alibaba Cloud Bailian platform.

The official address is as follows: https://bailian.console.aliyun.com/

After entering the official website, navigate to the model section, select the Flux model we want to experience, and we can choose either API call or page experience.

Here, the editor only conducted some simple tests on the page. It was found that the support for Chinese is really good, with understanding of Chinese supported, but rendering of Chinese is not very well supported. I hope that some big names in the open-source community will optimize the Chinese text rendering capabilities of Flux.1.